Entropy encoding, Hilbert space, and Karhunen-Loève transforms

Historically, the Karhunen-Loève KL decomposition arose as a tool from the interface of probability theory and information theory see details with references inside the paper . It has served as a powerful tool in a variety of applications, starting with the problem of separating variables in stochastic processes, say, Xt, and processes that arise from statistical noise, for example, from fractional Brownian motion. Since the initial inception in mathematical statistics, the operator algebraic contents of the arguments have crystallized as follows: start from the process Xt, for simplicity assume zero mean, i.e., E Xt =0, and create a correlation matrix C s , t =E XsXt . Strictly speaking, it is not a matrix, but rather an integral kernel. Nonetheless, the matrix terminology has stuck. The next key analytic step in the Karhunen-Loève method is to then apply the spectral theorem from operator theory to a corresponding self-adjoint operator, or to some operator naturally associated with C: hence, the name the Karhunen-Loève decomposition KLD . In favorable cases discrete spectrum , an orthogonal family of functions fn t in the time variable arise and a corresponding family of eigenvalues. We take them to be normalized in a suitably chosen square norm. By integrating the basis functions fn t against Xt, we get a sequence of random variables Yn. It was the insight of KL Ref. 25 to give general conditions for when this sequence of random variables is independent and to show that if the initial random process Xt is Gaussian, then so are the random variables Yn see also Example 3.1 below. In the 1940s, Karhunen pioneered the use of spectral theoretic methods in the analysis of time series and more generally in stochastic processes. It was followed up by the papers and books by Loève in the 1950s Ref. 25 and in 1965 by Ash. Note that this theory precedes the surge in the interest in wavelet bases! As we outline below, all the settings impose rather stronger assumptions. We argue how more modern applications dictate more general theorems, which we prove in our paper. A modern tool from operator theory and signal processing which we will use is the notion of frames in Hilbert space. More precisely, frames are redundant “bases” in Hilbert space. They are called framed but intuitively should be thought of as generalized bases. The reason for this, as we show, is that they